Heuristic Search of Optimal Reduction Schedule in Heterogeneous Cluster Environments

Network of workstation (NOW) is a coste ective alternative to massively parallel supercomputers. As commercially available o -theshelf processors become cheaper and faster, it is now possible to build a PC or workstation cluster that provides high computing power within a limited budget. However, a cluster may consist of di erent types of processors and this heterogeneity within a cluster complicates the design of efcient collective communication protocols, and it is a very hard combinatorial problem to design an optimal reduction protocol. Nevertheless, we show that a simple exchange lemma can greatly reduce seraph space so that we can nd the optimal solution e ciently. Combined with standard branchand-nound technique the search space is reduced to be less than one percent of original size for most practical cluster size.